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Abstract 

This paper introduces the application of the Template 
Matching algorithm in recognition dynamic hand gestures. 
Images undergo thresholding to convert each pixel of 
image into binary which will eventually be filtered to 
reduce the noise present. To make consistent and precise 
detection, this image can still be improved by enhancing 
the edge of the hand, which serves as the initial input for 
the process of template matching. The saved templates 
serve as the basis for comparison in Template Matching 
algorithm. The processed sequence of images will be 
compared to the templates saved in the database. When 
the current templates have significant relation or no 
significant relation to the saved database, the result will 
turn to the recognition of the mouse function. Hence, the 
mouse function will be executed. 

Keywords: template matching, dynamic hand gesture, 
virtual mouse 


RGB colours is used to perform mouse functions. Another 
paper entitled “Simulation of Real-time Hand Gesture 
Recognition for Physically Impaired” [9] uses Principal 
Component Analysis (PCA) by combining hand gesture 
recognition based on centroids with real-time 3D hand 
tracking, which achieved a recognition rate of 87.75%. A 
recent research for virtual mouse using static hand gesture 
recognition entitled “Cursor Control Using Hand Gestures 
and a Webcam” [10] that received a recognition rate of 
58.8%%. It used convexity defects of hand contour and 
analysed the fingertips and the angle of the fingers in order 
to perform the mouse functions. 

Based on the previous studies that used static gesture 
recognition, the tremor of the hand has become more 
noticeable when hanged in the air. Thus, due to the low 
precision detection of gesture caused by the instability of 
the hand, misdetection and inaccuracy of recognition had 
been the results. 


Nomenclature 

PCA 

RGB 

YCBCR 

3D 


Principal Component Analysis 

Red, Green and Blue 

Luminance; Chroma Blue; Chroma Red 

Three-dimensional 


1. Introduction 

The evolution of computer mouse has become a vigorous 
trend as technology develops rapidly. Yet, there have been 
many transformations of a computer mouse since the start 
of its invention, from a trackball to an optical mouse. 
Today’s generation has become more involved in creating 
devices that can replace mechanically operated 
technologies. One of the technologies that substituted the 
traditional computer mouse is called virtual mouse. This 
technology enables the user to navigate the window and 
execute functions of a mouse by only executing actions 
like static hand gestures. 

A research study entitled “Gesture Recognition Based 
Mouse Events” [13] that achieved 87.5% rate of 
recognition, involves the use of YCbCr color model. In 
this method, the dependency on light intensity has been 
eliminated. To track the gesture positions, selection of the 


The authors intend to provide a system that can recognize 
the dynamic hand gestures as mouse control. It integrates 
the use of dynamic hand gestures to eliminate the effect of 
tremor of the hand in recognizing the gesture. Embedded 
in the system are the templates saved in the database. 
These frames are composed of pictures which are captured 
from the starting position of the hand until the last position 
of the hand while performing a certain mouse function, 
served as the pilot copy in the database. The system 
integrates the method of template matching to increase the 
accuracy of the hand gesture recognition. As the camera 
detects a hand gesture, the captured input data will be 
processed through thresholding. 

Reduction of noise is also necessary by implementing 
image filtering and edge sharpening for the enhancement 
of the edges of the hand. Template matching of the saved 
templates and the current detected templates is the next 
process of the system. From the frames of the templates, 
the current template will be matched to the saved 
templates. If there is relation in the matching process, the 
gesture would be recognized. Thus, the mouse control with 
respect to the recognized gesture is performed. 
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The remained of the paper is organized as follows: Section 
(2) discusses the theory about Template Matching 
algorithm, how it works and how it is applied in the system 
to recognize dynamic hand gesture; Section (3) presents 
the results and discussion; and Section (4) presents the 
conclusion where the summary of the contribution of this 
work and identify areas for further work is presented. 

2. Theory 

The system for recognizing the dynamic hand gesture is 
equipped with the use of computer with 2.3 GHz 8GB 
RAM, administering Windows 7. This is made possible 
through the use of Microsoft Visual Studio 2012, which 
has embedded Aforge and Accord framework for C# .NET 
language, with MS Visual Studio form environment. The 
assessment of the result depends on the performance of the 
system. The set up for the distance of the hand and the 
camera is approximately 1 meter, depending on the 
external interferences. 

A. Recognizing dynamic hand gesture using Template 
Matching algorithm for controlling a virtual mouse 
Figure 1 illustrates how the system works step by step. It 
starts from how the camera captures the movement until 
the recognition of the dynamic hand gesture. After the 
camera detects the gesture, the captured input data would 
be processed through the method of segmentation to 
identify the correct contour. 

For the reduction of noise, filtering is necessary to have a 
clear image of the hand through smoothening, sharpening 
and edge enhancement. Once the data is easily 
distinguished, the process for gesture recognition 
proceeds. Template matching of the saved templates and 
the current detected templates is the next process of the 
system. 

Subsequently, this study involves the use of dynamic hand 
gestures; the data should include sequence of frames 
according to the posture of the hand. The saved templates 
would be used as the reference for comparison to the 
acquired data. This system integrated a program for mouse 
events. When there is correct matching, the system will 
automatically perform the mouse function. 

From the pre-processing segment of the system, the user’s 
way of moving his or her hand allows the system to either 
easily recognized the gesture or does not recognize the 
gesture performed. The detection of the system depends 
on how the environment affects the capturing of the hand, 
and how the user executes certain mouse function which 
also determines that the hand of the user must be of the 
same orientation as the saved templates. Moreover, this 
paper involves the application of the Template Matching 
algorithm in recognizing the dynamic hand gestures to 
make this system be more accurate in recognizing moving 
gestures. For the result of the system, the current templates 
have significant relation or no significant relation to the 
saved database. 



Figure 1. Conceptual Framework 



Figure 2. Hand Detection for HSV Color Model 


Hand Detection: This includes processes of acquiring data 
from the Template Matching algorithm. The raw data of 
the system are the frames or postures of the hand while 
performing a certain mouse function. The skin detection 
method is used to determine the hand area. The R, G and 
B pixels are used to find the skin regions that includes a 
wide range of colors such as red, pink, brown and orange. 
Training samples of skin region is done by saving a sample 
color. Once the hand is identified through skin detection, 
image segmentation is the next process to be done. It is 
applied to separate the arm region from the hand region. 
When the threshold process takes place, which involves 
replacing each pixel of an image into black pixel where the 
image intensity is lower than some set constant or a white 
pixel if it is higher than that set constant, the next processes 
are the noise reduction and sharpening of the hand to 
enhance the edges. 

Template Matching Algorithm: This process involves 
comparing and classifying the acquired data of the hand 
gesture to the saved templates. Since Template Matching 
is the involved process in performing the dynamic hand 
gesture, each gesture has its respective set of frames that 
are dependent on how the gesture is being done. 
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The concept of Template Matching is to provide a 
reference image and an image to be compared to the saved 
templates during the Template Matching process. The 
designated dynamic hand gesture will have certain time for 
training and saving the templates. The saved templates 
after the training will have 4 images that comprise the 
template. 



Figure 3. Hardware of the System 



Figure 4. Software Applied in the System 



Functions: Left click (left) and 
Right Click (right) 


't 




* 





Figure 6. Template for Scroll 
Functions: Scroll-up (left) 
and Scroll-down (right) 


The user will execute a dynamic hand gesture that has 
equivalent mouse function. The data will start from the 
frames or posture of the hand from the beginning up to the 
final position of the hand. The saved frames for each 
template will be compared to the current set of images. 
Correct recognition of the hand can only be possible if the 
pre-processes for hand detection are achieved correctly. 
When the templates to be compared are turned into binary 
(white and black), the process for template matching is 
also initialized. The user should conduct an experiment in 


which the system would compare the current templates of 
each dynamic hand gesture to the saved templates. 

Gesture Recognition : To compare current templates to the 
saved templates, posture and positioning of the fingers 
should be properly observed. Left click function is 
executed by bending the index finger, just like how a 
mechanical mouse is used. For right click, the middle 
finger should be slightly bent also. For the scroll up and 
down, the templates are illustrated. In drag and drop 
functions, index finger is also used. 

B. Evaluating the system’s performance in terms of 
accuracy and identifying the significant 
improvement of the current study in comparison to 
the previous studies 

This paper focuses on how to increase the accuracy rate of 
gesture recognition for mouse control and its significant 
improvement from the previous study. The scheme for 
assessing the system’s performance is that the accuracy of 
the hand based on the actual dynamic gesture and image 
template is checked by performing the mouse functions so 
that it can be proved that the instability of the hand will 
not affect the performance in terms of accuracy of the 
mouse functions since the gesture is based on dynamic 
movements. The method for evaluating the accuracy is to 
perform 30 trials for each mouse function. 

To compute for the accuracy rate of the system, 
recognition rate formula is involved. Number of 
recognized gestures will be provided from the experiment 
conducted by the proponents. To know the error rate of the 
system, number of unrecognized gestures over the total 
number of trials is the formula to be used. 

Recognition Rate 

Number of Recoqnized Gestures 

= x 100 

Number of Trials for Each Mouse Function 

a) 


Error Rate 

Number of Unrecognized Gestures 
Number of Trials for Each Mouse Function 

( 2 ) 

This study provides a system that can recognize a certain 
dynamic hand gesture that has corresponding mouse 
function. It involves six mouse functions such as left click, 
right click, double click, scroll up, scroll down and drag 
and drop. The accuracy rate is the implicated parameter 
which needs to be improved by the current study. The 
result of the current study will be compared to the data of 
the previous study provided below and it will be used to 
identify the significant improvement of the current study 
from the previous studies. 


Table 1 . Result from the Study Entitled “Gesture 


Recognition Based Mouse Events” 


Mouse Functions 

Detection Rate 

Right Click 

92% 

Left Click 

87% 

Double Click 

87% 

Drag and Drop 

84% 
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Table 2. Result from the Study Entitled “Simulation of 
Real-time Hand Gesture Recognition for Physically 
Impaired” 


Mouse Functions 

Detection Rate 

Right Click 

88% 

Left Click 

91% 

Scroll Down 

87% 

Scroll Up 

85% 


Table 3. Result from the Study Entitled “Cursor Control 
System Using Hand Gesture Recognition” 


Mouse Functions 

Basis 

Right Click 

13/30 

Left Click 

11/30 

Double Click 

16/30 

Scroll Down 

3/30 

Scroll Up 

19/30 


3. Results and Discussion 

The study aims to provide a system that can achieve a high 
accuracy rate in terms of recognizing the dynamic hand 
gesture for each mouse movement. The authors conducted 
the gathering of data by allowing 30 trials for each mouse 
function to be executed by the user. 


Table 4, Result of the Study 


Mouse Functions 

Recognized 

Missed 

Left Click 

30/30 

0/30 

Right Click 

28/30 

2/30 

Double Click 

25/30 

5/30 

Scroll Up 

30/30 

0/30 

Scroll Down 

30/30 

0/30 

Drag and Drop 

28/30 

2/30 


Some of the samples exhibiting recognition errors or 
misclassified errors are shown in Table 4. In the 
experiment that was conducted by the proponents, the 
mismatched mouse functions are as follows: right click, 
double click, and drag and drop. 

Table 5. Mismatched and Unrecognized Gestures 



Table 5 indicates the mismatched templates and 
unrecognized gestures from the gathered data. Due to the 
variation of finger positions in the frames extracted from 
the real-time video, it lead to erroneous detection of the 
hand features that caused the system to misinterpret the 
gesture being performed by the user. The last position of 
the gesture performed is incorrectly recognized shown in 
Table 5, since double click function is set as two 
consecutive left click function like a mechanical mouse, 
the user is likely to perform the double click function 
immediately without a pause that lead to recognition of 
incorrect gestures. The function drag and drop is not 
properly recognized due to the improper orientation of the 
hand in the last position of the frames extracted from the 
real-time video. 


Table 6. Accuracy Rate and Error Rate of the System 


Mouse Functions 

Accuracy Rate 

Error Rate 

Left Click 

100% 

0% 

Right Click 

93.3333% 

6.6667% 

Double Click 

83.3333% 

16.6667% 

Scroll Up 

100% 

0% 

Scroll Down 

100% 

0% 

Drag and Drop 

93.3333% 

6.6667% 


Table 6 indicates the accuracy rate and error rate of the 
system’s performance. The accuracy rate is computed 
using the formula for the recognition rate. Recognition rate 
is just the ratio of success recognitions to the total number 
of trials. The system’s performance in terms of accuracy 
for the left click, scroll up and scroll down is 100%. There 
is 93.3333% for the mouse functions right click, and drag- 
and-drop. For the double click function, there is 83.3333% 
of recognition rate. 

The tables below show the significant improvement of 
the current study from the previous studies. 


Table 7. Comparison of Recognition Rate of the Current 
Study from the Study Entitled “Gesture Recognition 
Based Mouse Events” 


Mouse Functions 

Current Study 

Previous Study 

Left Click 

100% 

87% 

Right Click 

93.3333% 

92% 

Double Click 

83.3333% 

87% 

Drag and Drop 

93.3333% 

84% 

Overall 

92.5% 

87.5% 

Recognition Rate 


Table 8. Comparison of Recognition Rate of the Current 
Study from the Study Entitled “Simulation of Real-time 
Hand Gesture Recognition for Physically Impaired” 


Mouse Functions 

Current Study 

Previous Study 

Left Click 

100% 

91% 

Right Click 

93.3333% 

88% 

Scroll Up 

100% 

85% 

Scroll Down 

100% 

87% 

Overall 

Recognition Rate 

98.3333% 

87.75% 
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Table 9. Comparison of Recognition Rate of the Current 
Study from the Study Entitled “Cursor Control System 
using Hand Gesture Recognition” 


Mouse Functions 

Current Study 

Previous Study 

Left Click 

100% 

63% 

Right Click 

93.3333% 

57% 

Double Click 

83.3333% 

47% 

Scroll Up 

100% 

37% 

Scroll Down 

100% 

90% 

Overall 

Recognition Rate 

95.33332% 

58.8% 


In identifying the system’s performance in terms of its 
accuracy, the authors executed 30 trials for each mouse 
function which is based on the reference journal. The 
current system is able to identify six (6) mouse functions, 
while the previous system has 5 mouse functions. This 
study has recognition rate of 83 to 100 percent for each 
mouse function. This numerical value signifies that there 
is only low number of missed recognition for each mouse 
function with error rate of 5.0002%. 

The authors assess the result of the experiments by 
observing the possible reasons of the missed recognition. 
From the gathered data, it can be seen that there are several 
reasons for the misclassification of the gestures. The 
authors evaluate that some misclassified gestures are 
caused by disorientation because of the tilting or slanting 
of the user’s hand while executing a function (drag and 
drop). It is also show that when the gesture is not 
performed well from the start of the execution, 
misclassification of the hand gesture occurs which affects 
the recognition of the system. This happens when the 
system (camera) is not able to determine that the user 
already moved her finger, in case of the right click 
function. 

This study aims to compare the result with previous 
studies. As to compare the results from the previous study, 
the overall system’s performance with regards to accuracy 
is computed based on the five mouse functions. The 
previous study achieved a rate of recognition of about 
58.8% while the current study has overall recognition rate 
of 95.33332%. 

The authors also compare the result of the experiment 
from other two papers. The results of the past studies are 
87.75% and 87.5%. The result of the current study implies 
that there is significant improvement in terms of accuracy 
of the system. 

4. Conclusion 

Recognition of the dynamic hand gesture is carried upon 
the implementation of Template Matching in executing 
mouse functions. The designated dynamic hand gestures 
for the mouse functions are unique, so that it will not cause 
confusion to the system. The saved template is used as a 
basis for comparing the actual template executed by the 
user and the existing template. 

The authors find that hand detection process of the system 
highly affects the recognition process of the dynamic hand 
gesture because the pre-processes of the system should be 
completed properly to make the actual templates have less 
noise and be ready for recognition process. This means 
that when the detection is not properly working, process of 


matching will be affected. This detection is affected by the 
lighting condition of the place where it is performed. The 
proponents are able to arrive with a system that will 
manage proper recognition of dynamic gestures that are 
the same in controlling a commercialized mouse. 

The authors conclude that the current system surpassed the 
system’s performance of the previous studies. This study 
also proves that the problem of inaccuracy due to tremor 
of the hand has been eliminated by integrating Template 
Matching algorithm in dynamic hand gestures. 
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